Skip to content

Improve ql-mcp VS Code extension UX#230

Open
Copilot wants to merge 11 commits intomainfrom
copilot/improve-vscode-extension-ux
Open

Improve ql-mcp VS Code extension UX#230
Copilot wants to merge 11 commits intomainfrom
copilot/improve-vscode-extension-ux

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 7, 2026

Closes #229.

Summary of Changes

This PR introduces several improvements to the VS Code extension for the ql-mcp server -- focusing on robustness, logging clarity, and test coverage in order to improve the overall user experience (UX) with the CodeQL MCP VS Code extension. The most significant changes include enhanced concurrency handling and validation in CLI resolution, improved logging and debouncing for server and pack operations, and expanded test cases. Additionally, file path logging is now more user-friendly, and the test infrastructure is strengthened for watcher events.

Outline of Changes

CLI Resolution and Robustness Improvements:

  • Refactored CliResolver to ensure only one resolution is in progress at a time, preventing duplicate work and race conditions. Added a resolvePromise field and split logic into a new doResolve method. [1] [2]
  • Improved CLI detection: when resolving via PATH, the binary is now validated before use, and a warning is logged if validation fails.
  • Added test coverage for CLI version detection and concurrent resolution scenarios.
  • Ensured cache invalidation clears in-flight resolution promises.

Logging and Diagnostics Enhancements:

  • Updated logging in PackInstaller to provide detailed status messages about detected CLI versions, pack downloads, and bundled pack installation results, including per-language success counts. [1] [2] [3] [4]
  • Changed file event logging in DatabaseWatcher and QueryResultsWatcher to use relative paths for improved readability. [1] [2] [3]
  • Improved bundled server log message for clarity.

Server Notification and Watcher Behavior:

  • Reworked watcher-triggered environment update logic: file-content changes no longer trigger MCP server definition changes, as the server discovers files itself; watchers now focus on logging and internal state.
  • Introduced debounce logic to McpProvider.fireDidChange to coalesce rapid-fire notifications into a single event, and ensured restarts cancel pending notifications. [1] [2] [3]

Testing and Mocks:

  • Added new integration tests to the test suite.
  • Enhanced test mocks for DatabaseWatcher and QueryResultsWatcher to support manual firing of change events. [1] [2]
  • Updated test imports for new watcher classes.

API and Utility Updates:

  • Added asRelativePath and updateWorkspaceFolders mocks to the VS Code mock workspace API.

These changes collectively improve the extension’s reliability, observability, and testability.

Copilot AI and others added 4 commits April 7, 2026 04:19
Add completable() wrappers to workflow prompt parameters so VS Code
Copilot Chat shows auto-complete dropdowns for language, query paths,
SARIF files, database paths, and pack roots.

- Create prompt-completions.ts with completion providers for each
  parameter type (language enum, .ql/.qlref files, .sarif files,
  CodeQL databases, codeql-pack.yml directories)
- Update all 14 workflow prompt registrations to use addCompletions()
- Add comprehensive unit tests (35 tests covering all completers
  and the addCompletions utility)

Agent-Logs-Url: https://github.com/advanced-security/codeql-development-mcp-server/sessions/e316ece4-06c5-45d7-8020-062e6ec67e39

Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com>
Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com>
- cloneStringType now accepts both ZodString and ZodEnum (for prompts
  that use raw schema.shape without toPermissiveShape)
- Add runtime type checks that throw clear errors for unexpected types
- Add 2 unit tests for ZodEnum handling in addCompletions

Agent-Logs-Url: https://github.com/advanced-security/codeql-development-mcp-server/sessions/e316ece4-06c5-45d7-8020-062e6ec67e39

Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com>
Copilot AI changed the title [WIP] Improve UX for ql-mcp workflow prompts in VS Code [UPDATE PRIMITIVE] Add MCP prompt argument completions for VS Code slash command UX Apr 7, 2026
Copilot AI requested a review from data-douser April 7, 2026 04:35
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 7, 2026

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Snapshot Warnings

⚠️: No snapshots were found for the head SHA 0bf6946.
Ensure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice.

Scanned Files

None

Addresses three UX issues with MCP prompt argument completions:

1. Database discovery: completeDatabasePath now scans $HOME/codeql/databases/
   as a default search location, and recursively discovers directories
   containing codeql-database.yml (including .testproj dirs) in the workspace.

2. Language auto-derivation: For prompts where queryPath is provided,
   language is now optional and auto-derived from the nearest
   codeql-pack.yml's codeql/<lang>-all or codeql/<lang>-queries dependency.
   Schema field ordering now puts queryPath before language so VS Code
   prompts for the query first. Affected schemas: explainCodeqlQuerySchema,
   documentCodeqlQuerySchema, workshopCreationWorkflowSchema,
   qlLspIterativeDevelopmentSchema.

3. Query path completion filtering: completeQueryPath now skips .github,
   dist, coverage, and build directories in addition to node_modules, .git,
   and .tmp — preventing workshop examples and build artifacts from
   cluttering the VS Code dropdown.

New tests: 19 unit tests covering all three fixes.
Updated tests: 9 existing tests updated to reflect language being optional.
Copilot AI review requested due to automatic review settings April 7, 2026 15:35
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds MCP prompt-argument completions so VS Code Copilot Chat slash commands can provide dropdown suggestions (languages, query paths, SARIF paths, databases, pack roots), and updates several prompt schemas/handlers to allow omitting language with best-effort auto-derivation from codeql-pack.yml. The PR also includes a set of VS Code extension-side changes around CodeQL CLI discovery and pack installation logging, plus new extension E2E coverage for completion/complete.

Changes:

  • Introduce server/src/prompts/prompt-completions.ts with completion providers and addCompletions() to wrap prompt shapes with MCP SDK completable().
  • Update workflow prompt registrations to apply addCompletions() and (for some prompts) make language optional + attempt auto-derivation from pack metadata.
  • Add unit tests (server) and E2E integration tests (VS Code extension) for completion behavior, alongside additional extension improvements/tests for CLI resolution and pack installer logging.
Show a summary per file
File Description
server/src/prompts/prompt-completions.ts New completion providers + addCompletions() wrapper; also adds language auto-derivation helper.
server/src/prompts/workflow-prompts.ts Wrap prompt registration shapes with addCompletions(); make language optional for several schemas; add handler-side language auto-derivation + warnings.
server/test/src/prompts/prompt-completions.test.ts New unit tests for completion providers, schema wrapping behavior, and language auto-derivation from pack metadata.
server/test/src/prompts/workflow-prompts.test.ts Update schema expectation tests to reflect language becoming optional in select prompts and updated handler messaging.
server/dist/codeql-development-mcp-server.js Updated bundled build output reflecting server-side changes.
extensions/vscode/test/suite/mcp-completion-e2e.integration.test.ts New Extension Development Host E2E tests calling completion/complete against the running MCP server.
extensions/vscode/test/suite/mcp-server.integration.test.ts Adds an environment validation test for CODEQL_PATH (currently substring-based).
extensions/vscode/esbuild.config.js Registers the new extension integration test entry point for compilation.
extensions/vscode/src/codeql/cli-resolver.ts Adds in-flight resolve de-duplication and validates PATH-resolved binaries before accepting them.
extensions/vscode/test/codeql/cli-resolver.test.ts Adds tests for PATH-based version detection and concurrent resolution behavior.
extensions/vscode/src/server/pack-installer.ts Enhances logging (CLI vs target version), adds success counts, and changes terminology to “install” for pack download.
extensions/vscode/test/server/pack-installer.test.ts Adds tests asserting updated pack installer logging and summary behavior.
extensions/vscode/src/server/server-manager.ts Tweaks “bundled server” log message text.
extensions/vscode/test/server/server-manager.test.ts Updates expectations to match new server-manager logging and asserts no “npm install” mention.

Copilot's findings

Comments suppressed due to low confidence (3)

server/src/prompts/workflow-prompts.ts:348

  • The explainCodeqlQuerySchema parameter comment still says queryPath and language are required, but language is now .optional(). Please update the comment so it matches the schema/handler behavior (language may be omitted and auto-derived).
 * - `databasePath` is optional – a database may also be derived from tests.
 */
export const explainCodeqlQuerySchema = z.object({
  queryPath: z
    .string()
    .describe('Path to the CodeQL query file (.ql or .qlref)'),
  language: z
    .enum(SUPPORTED_LANGUAGES)
    .optional()
    .describe('Programming language of the query (auto-derived from pack metadata when omitted)'),

server/src/prompts/workflow-prompts.ts:367

  • The documentCodeqlQuerySchema comment says queryPath and language are required, but language is now optional. Please update the comment to reflect the new validation contract (and the handler’s auto-derivation/warning behavior).
/**
 * Schema for document_codeql_query prompt parameters.
 *
 * - `queryPath` and `language` are **required**.
 */
export const documentCodeqlQuerySchema = z.object({
  queryPath: z
    .string()
    .describe('Path to the CodeQL query file (.ql or .qlref)'),
  language: z
    .enum(SUPPORTED_LANGUAGES)
    .optional()
    .describe('Programming language of the query (auto-derived from pack metadata when omitted)'),

server/src/prompts/workflow-prompts.ts:454

  • qlLspIterativeDevelopmentSchema now has language optional, but the schema comment still states language and queryPath are required. Please update the comment to match the schema (language omitted -> auto-derive + warning).
/**
 * Schema for ql_lsp_iterative_development prompt parameters.
 *
 * - `language` and `queryPath` are **required** – LSP tools need both.
 * - `workspaceUri` is optional – defaults to the pack root.
 */
export const qlLspIterativeDevelopmentSchema = z.object({
  queryPath: z
    .string()
    .describe('Path to the query file being developed'),
  language: z
    .enum(SUPPORTED_LANGUAGES)
    .optional()
    .describe('Programming language for the query (auto-derived from pack metadata when omitted)'),
  • Files reviewed: 13/15 changed files
  • Comments generated: 9

File watcher events (database discovery, query result creation) were
triggering envBuilder.invalidate() + mcpProvider.fireDidChange(), causing
VS Code to re-query provideMcpServerDefinitions() on every workspace
file change. The debounce only coalesced rapid events but did not prevent
the unnecessary re-provision cycle.

The MCP server definition only needs to change when:
- The extension itself changes (update/reinstall)
- Workspace folder registration changes (folders added/removed)
- Configuration changes that affect the server

The running server discovers databases and query results on its own
through filesystem scanning at tool invocation time, so notifying
VS Code of file content changes is unnecessary.

Also includes:
- Prompt completion caching with 5s TTL to avoid repeated filesystem
  scans during rapid completion requests
- Debounce timer cancellation when McpProvider is disposed
- Unit tests verifying watchers do not trigger fireDidChange
- Integration tests verifying file creation does not fire
  onDidChangeMcpServerDefinitions
@data-douser data-douser changed the title [UPDATE PRIMITIVE] Add MCP prompt argument completions for VS Code slash command UX Improve ql-mcp VS Code extension UX Apr 7, 2026
@data-douser data-douser requested a review from Copilot April 7, 2026 20:33
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves the ql-mcp VS Code extension and server prompt UX by adding MCP prompt argument completions, refining watcher/restart behavior, strengthening CLI resolution robustness, and expanding automated test coverage.

Changes:

  • Add prompt argument completion providers (language + file/path discovery) and wire them into workflow prompt registrations.
  • Make several workflow prompt language parameters optional with auto-derivation from CodeQL pack metadata and updated handler messaging/tests.
  • Improve extension-side robustness/UX via CLI resolution concurrency control, debounced MCP definition change notifications, clearer logging, and new unit/integration tests.
Show a summary per file
File Description
server/src/prompts/prompt-completions.ts Adds completion callbacks + schema enhancement via completable() and language auto-derivation from pack metadata.
server/src/prompts/workflow-prompts.ts Makes language optional for several prompts, derives language from pack metadata, and applies addCompletions() to prompt shapes.
server/test/src/prompts/prompt-completions.test.ts Adds unit tests covering completion behavior, caching, schema enhancement, and pack-language derivation.
server/test/src/prompts/workflow-prompts.test.ts Updates schema/handler tests for optional language and updated warning behavior.
server/dist/codeql-development-mcp-server.js Updates bundled build artifact to include the new prompt completion functionality and schema changes.
extensions/vscode/src/codeql/cli-resolver.ts Adds in-flight resolution deduplication and validation for PATH-discovered CLI binaries.
extensions/vscode/test/codeql/cli-resolver.test.ts Adds tests for PATH version detection, validation failure fallback, and concurrent resolution behavior.
extensions/vscode/src/server/mcp-provider.ts Debounces fireDidChange() notifications and cancels pending debounce on restart.
extensions/vscode/test/server/mcp-provider.test.ts Adds unit tests validating debounce/coalescing behavior and restart cancellation semantics.
extensions/vscode/src/extension.ts Stops invalidating/re-providing MCP definitions on file-content watcher events; keeps watchers for logging/state only.
extensions/vscode/test/extension.test.ts Adds tests ensuring watcher events do not trigger MCP definition updates or env invalidation.
extensions/vscode/src/server/server-manager.ts Clarifies bundled server log message (“Bundled server ready”).
extensions/vscode/test/server/server-manager.test.ts Updates assertions to match the new bundled server log wording and ensures no “npm install” mention.
extensions/vscode/src/server/pack-installer.ts Improves logging around CLI/target versions, download/install flow, and per-language success summaries.
extensions/vscode/test/server/pack-installer.test.ts Adds tests for new logging behaviors and summary counts.
extensions/vscode/src/bridge/database-watcher.ts Logs discovered/removed databases using relative paths for readability.
extensions/vscode/src/bridge/query-results-watcher.ts Logs created BQRS/SARIF results using relative paths for readability.
extensions/vscode/__mocks__/vscode.ts Extends workspace mock with asRelativePath and updateWorkspaceFolders for tests.
extensions/vscode/test/suite/mcp-server.integration.test.ts Adds an integration test asserting CODEQL_PATH shape when available.
extensions/vscode/test/suite/mcp-completion-e2e.integration.test.ts Adds end-to-end tests that exercise MCP completion/complete against the real server in the extension host.
extensions/vscode/test/suite/file-watcher-stability.integration.test.ts Adds integration tests ensuring file changes do not trigger redundant MCP definition change events.
extensions/vscode/esbuild.config.js Registers the new VS Code integration test entry points for esbuild compilation.

Copilot's findings

Comments suppressed due to low confidence (3)

server/src/prompts/workflow-prompts.ts:1034

  • The warning advises ensuring a codeql/<lang>-all dependency, but resolveLanguageFromPack() also resolves languages from codeql/<lang>-queries. To avoid misleading guidance, update the message to mention both -all and -queries (or align the resolver + message to the same rule).
        if (!effectiveLanguage) {
          warnings.push('⚠ **Language could not be auto-derived.** Please provide the `language` parameter or ensure the query is inside a CodeQL pack with a `codeql/<lang>-all` dependency.');
        }

server/src/prompts/workflow-prompts.ts:1098

  • The warning advises ensuring a codeql/<lang>-all dependency, but resolveLanguageFromPack() also resolves languages from codeql/<lang>-queries. To avoid misleading guidance, update the message to mention both -all and -queries (or align the resolver + message to the same rule).
        if (!effectiveLanguage) {
          warnings.push('⚠ **Language could not be auto-derived.** Please provide the `language` parameter or ensure the query is inside a CodeQL pack with a `codeql/<lang>-all` dependency.');
        }

server/src/prompts/workflow-prompts.ts:1215

  • The warning advises ensuring a codeql/<lang>-all dependency, but resolveLanguageFromPack() also resolves languages from codeql/<lang>-queries. To avoid misleading guidance, update the message to mention both -all and -queries (or align the resolver + message to the same rule).
        if (!effectiveLanguage) {
          warnings.push('⚠ **Language could not be auto-derived.** Please provide the `language` parameter or ensure the query is inside a CodeQL pack with a `codeql/<lang>-all` dependency.');
        }
  • Files reviewed: 21/23 changed files
  • Comments generated: 5

@data-douser data-douser marked this pull request as ready for review April 7, 2026 21:43
@data-douser data-douser requested review from a team and enyil as code owners April 7, 2026 21:43
Copilot AI review requested due to automatic review settings April 7, 2026 21:43
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves the ql-mcp VS Code extension and MCP server prompt UX by adding prompt-argument completions (for better slash-command input), making several workflow prompt parameters more robust (e.g., optional language with auto-derivation), and reducing noisy/redundant extension definition-change events via watcher behavior changes and debouncing.

Changes:

  • Add MCP prompt argument completion providers (language + workspace file/path discovery) and wire them into workflow prompt registration.
  • Make language optional for several workflow prompts and auto-derive it from codeql-pack.yml when possible, emitting warnings when it can’t be derived.
  • Improve VS Code extension robustness: CLI resolution concurrency/validation, pack install/download logging, watcher behavior, and debounced MCP definition-change notifications; expand unit/integration test coverage accordingly.
Show a summary per file
File Description
server/test/src/prompts/workflow-prompts.test.ts Updates prompt schema/handler tests to reflect optional language and new warning behavior.
server/test/src/prompts/prompt-completions.test.ts Adds unit tests for completion providers, caching, directory filtering, and pack-language derivation.
server/test/src/prompts/constants.test.ts Adds tests ensuring shared SUPPORTED_LANGUAGES constant is exported and sorted.
server/src/prompts/workflow-prompts.ts Makes language optional for select prompts, auto-derives language from pack metadata, and attaches completion metadata to prompt schemas.
server/src/prompts/prompt-completions.ts Introduces completion providers (language/query/db/sarif/packRoot) plus schema enhancement via completable().
server/src/prompts/constants.ts Moves shared prompt constants (supported languages) to a dedicated module to avoid circular deps.
server/dist/codeql-development-mcp-server.js Updates bundled server output to include new prompt completion and prompt behavior changes.
extensions/vscode/test/suite/mcp-server.integration.test.ts Adds an integration assertion around CODEQL_PATH shape when present.
extensions/vscode/test/suite/mcp-completion-e2e.integration.test.ts Adds end-to-end tests that invoke MCP completion/complete for prompt arguments.
extensions/vscode/test/suite/file-watcher-stability.integration.test.ts Adds integration coverage ensuring file changes don’t spuriously trigger definition updates.
extensions/vscode/test/server/server-manager.test.ts Updates expectations for revised “bundled server ready” logging.
extensions/vscode/test/server/pack-installer.test.ts Adds tests for new pack install/download logging and success-count summaries.
extensions/vscode/test/server/mcp-provider.test.ts Adds unit tests for debounced fireDidChange, restart behavior, and disposal.
extensions/vscode/test/extension.test.ts Adds tests ensuring watcher events don’t trigger environment invalidation or definition change events.
extensions/vscode/test/codeql/cli-resolver.test.ts Adds tests for PATH validation/version detection, concurrent resolution, and invalidation of in-flight resolve.
extensions/vscode/src/server/server-manager.ts Refines bundled-server readiness log message for clarity.
extensions/vscode/src/server/pack-installer.ts Enhances logging (CLI/target versions, download vs bundled install, per-language success counts).
extensions/vscode/src/server/mcp-provider.ts Debounces fireDidChange, cancels pending debounce on restart/dispose.
extensions/vscode/src/extension.ts Stops treating watcher file events as a trigger for MCP server definition changes (logging/state only).
extensions/vscode/src/codeql/cli-resolver.ts Prevents duplicate concurrent CLI resolutions, validates PATH results, and discards stale in-flight results after invalidation.
extensions/vscode/src/bridge/query-results-watcher.ts Logs watcher events using relative paths for readability.
extensions/vscode/src/bridge/database-watcher.ts Logs discovered/removed databases using relative paths for readability.
extensions/vscode/esbuild.config.js Adds new integration test entry points to the extension test build config.
extensions/vscode/mocks/vscode.ts Extends VS Code workspace mock with asRelativePath and updateWorkspaceFolders.

Copilot's findings

  • Files reviewed: 23/25 changed files
  • Comments generated: 3

@data-douser
Copy link
Copy Markdown
Collaborator

@copilot apply changes based on the comments in this thread

…EffectiveLanguage helper, add Windows env stubs

- Centralize directory skip list into SKIP_DIRS set used by
  findFilesByExtension, findDatabaseDirs, and completePackRoot
  (adds dist/, coverage/, build/, .github/ to completePackRoot)
- Extract getEffectiveLanguage() helper to eliminate duplicated
  language auto-derivation logic across 4 prompt handlers
- Add Windows env var stubs (HOMEDRIVE, HOMEPATH, USERPROFILE)
  to database completion tests for cross-platform compatibility
- Add 5 unit tests for getEffectiveLanguage()

Agent-Logs-Url: https://github.com/advanced-security/codeql-development-mcp-server/sessions/e315dcb4-ed12-428b-8294-14b398746c4c

Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 7, 2026

@copilot apply changes based on the comments in this thread

All three review comments addressed in 63247cc:

  1. completePackRoot() skip list — Centralized all directory skip lists into a shared SKIP_DIRS set, so completePackRoot() now skips dist/, coverage/, build/, .github/ (matching findFilesByExtension() and findDatabaseDirs()).

  2. Windows env var stubs — Added HOMEDRIVE, HOMEPATH, and USERPROFILE stubs to the completeDatabasePath default search path tests for cross-platform compatibility.

  3. Duplicated language auto-derivation — Extracted getEffectiveLanguage() helper in prompt-completions.ts and replaced all 4 duplicated instances in workflow-prompts.ts. Added 5 unit tests for the new helper.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve vscode extension UX for ql-mcp workflow prompts

3 participants